System and method for multiclass classification of images using a programmable light source

ABSTRACT

An apparatus, system and process for identifying one or more different tissue types are described. The method may include applying a configuration to one or more programmable light sources of an imaging system, where the configuration is obtained from a machine learning model trained to distinguish between the one or more different tissue types captured in image data. The method may also include illuminating a scene with the configured one or more programmable light sources, and capturing image data that includes one or more types of tissue depicted in the image data. Furthermore, the method may include analyzing color information in the captured image data with the machine learning model to identify at least one of the one or more different tissue types in the image data, and rendering a visualization of the scene from the captured image data that visually differentiates tissue types in the visualization.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.15/445,121, filed Feb. 28, 2017, which is hereby incorporated byreference in its entirety.

TECHNICAL FIELD

This disclosure relates generally to digital imaging, and in particularbut not exclusively, relates to classification of objects within imagedata.

BACKGROUND INFORMATION

When an image of an object is captured by a camera, a user may want toenhance the image by adjusting contrast of the image, accentuatingcertain features of the image, etc. when analyzing the object capturedwithin the camera's field of view. For example, an image may be enhancedto distinguish between different mineral/soil compositions (e.g., inaerial or ground photography), to highlight blood flow (e.g., oxygenatedand non-oxygenated blood), to perform emotion recognition and/ordetection, to distinguish anatomy in surgery (e.g., to distinguishbetween benign and malignant tissue), etc. In medical imaging inparticular, enhancing medical images, for example to distinguish betweenbenign and malignant tissue, increases the safety and expands the reachof surgical interventions.

Specifically, during laparoscopic surgery, a surgeon inserts anendoscope into a small incision of a patient. The endoscope illuminatesand captures images of a surgical area in real-time while the surgeonperforms the surgery through the same or other incision, typicallywithout physically viewing the area being operated on. A significantproblem with such surgeries and the imaging used during such surgeriesis that of low contrast between tissue(s) of interest and othertissue(s). For example, in a surgical procedure to excise canceroustissue, the surgeon wants high contrast between the cancerous tissue andthe healthy tissue so that the cancerous tissue can be removed whilemaking no/minimal damage to surrounding healthy tissue. The problem iseven more acute in surgery close to nerve tissue, where high contrastbetween adipose, cancerous, or other tissue and nerve tissue is neededto prevent inadvertent contact and/or removal of nerve tissue to preventdamage to the nervous system.

One solution to distinguishing between different tissue types duringsurgical procedures includes injecting fluorescent dyes into a patientwhere different dyes bind to different tissue types. When imaged withillumination from an excitation laser, the dye will fluoresce, and thedesired tissue will appear much brighter in captured image data. Toobtain high contrast between multiple tissues, a specific dye with adifferent excitation wavelength for each type of tissue is needed, adifferent excitation laser is needed for each tissue type, and aseparate image for each tissue type is also needed. This approach fordistinguishing between tissue types, however, typically requires the useof toxic dyes having specific fluorescence, which have not been approvedby the Food and Drug Administration for use in humans, the developmentof which is a long, costly process. Furthermore, specific wavelengthexcitation lasers are needed and images must be captured for eachdye/tissue combination. Thus, to obtain multiclass contrast betweendifferent types of tissues, the number of lasers and the number ofimages scales linearly with the number of tissue types to bedifferentiated.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the invention aredescribed with reference to the following figures, wherein likereference numerals refer to like parts throughout the various viewsunless otherwise specified. The drawings are not necessarily to scale,emphasis instead being placed upon illustrating the principles beingdescribed.

FIG. 1 is a block diagram of an exemplary system architecture formulticlass classification of images using a programmable light source.

FIG. 2 is a block diagram of one embodiment of an endoscope and an imageprocessing device having an endoscope controller.

FIG. 3 illustrates one embodiment of a process for using a machinelearning model to perform multiclass classification of images using aprogrammable light source.

FIG. 4 is a flow chart illustrating one embodiment of a process fortraining a machine learning model for multiclass classification ofimages using a programmable light source.

FIG. 5 is one embodiment of a computer system that may be used with thepresent invention.

FIG. 6 illustrates one embodiment of a topography of a machine learningmodel for performing multiclass classification between different tissuetypes.

FIG. 7 illustrates one embodiment of exemplary inputs for determiningrecorded color pixel values.

DETAILED DESCRIPTION

Embodiments of an apparatus, system, and process for performingmulticlass classification of objects within images using a programmablelight source are described herein. In embodiments, a programmable lightsource illuminates a scene and a camera captures image data of theilluminated. In one embodiment, the programmable light source and thecamera are integrated into an endoscope for use during, for example,laparoscopic surgery being performed by a medical professional.

In one embodiment, image data captured by the camera is provided to atrained machine learning model, such as a trained neural network, thatanalyzes color values of individual pixels of the image data to classifythe individual pixels as belonging to one or more different classes. Inembodiments, the different classes can be different classes of tissueimaged during a medical procedure. Furthermore, as discussed in greaterdetail below, the color values of the individual pixels are a functionof the spectrum of illumination provided by the programmable lightsource, spectral reflectivities of tissue(s) being imaged, and colorfunctions of a filter of the camera (e.g., Bayer filter colorfunctions). The machine learning model may be trained prior to use usingmultispectral imaging and samples of classes of tissue to determine aspectrum of illumination of the programmable light source that optimallydistinguishes between the classes of tissue. Furthermore, during use,the color values of the individual pixels are provided to the trainedmachine learning model to distinguish the classes of tissue on aper-pixel basis. Beneficially, hyperspectral quality imaging is used totrain the machine learning model before use, but a much less bulky andmuch faster discrete imaging system is utilized during use to classifydifferent tissue types based on color pixel values.

In embodiments, based on the classification of the pixels of thecaptured image data, the different classes of image data can bedistinguished when rendered to a user, such as a surgeon performing amedical procedure. For example, a rendered image of a scene can includecolor overlays that visually distinguish between different tissue types.In embodiments, the surgeon can provide feedback, such as highlightingareas believed to be certain tissue types. This feedback data can beprovided back to the machine learning model for training a machinelearning model in real time and/or refinement of an existing machinelearning model.

The description below refers to an endoscope with a camera and aprogrammable light source used during medical procedures fordistinguishing between different classes of tissue. However, thepresently described embodiments are not limited to endoscopes and/ormedical procedures. Rather, the embodiments and techniques describedherein are flexible, and can be used for performing multiclassclassification for a wide class of imaging applications in which typesof objects within image data are to be classified. However, for sake ofclarity, the below description will be focused on multi-class tissuetype classification.

Some portions of the detailed description that follow are presented interms of algorithms and symbolic representations of operations on databits within a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “applying”, “illuminating”, “capturing”, “analyzing”,“rendering”, “determining”, “identifying”, “generating”, “measuring”,“using”, “receiving”, “providing”, or the like, refer to the actions andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical (e.g.,electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The present invention also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but not limited to, any type of diskincluding floppy disks, optical disks, CD-ROMs, and magnetic-opticaldisks, read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, or any type of media suitable forstoring electronic instructions.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct a more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear from the description below.In addition, the present invention is not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof the invention as described herein.

FIG. 1 is a block diagram of an exemplary system architecture formulticlass classification of images using a programmable light source.In one embodiment, the system 100 includes an endoscope 110 communicablycoupled with an image processing device 120. In embodiments, endoscope110 may be coupled via physical connection (e.g., wired connection), viaa wireless connection (e.g., wireless network, personal area network,near field communication, etc.), or other type of communications link.Furthermore, the image processing device 120 may be communicably coupledwith a display 130 and an imaging data store 140, over any of thecommunications links discussed herein.

Endoscope 110 is responsible for capturing images of a scene with camera114. Camera 114 may include a lens 116 and an image sensor 118. The lens116 of camera 114 allows light to pass from outside of endoscope 110(e.g., a scene) to the image sensor 118 of camera 114. In oneembodiment, the image sensor is a Bayer image sensor that includes acolor arrangement of red, green, and blue color filters on a pixel arrayof the image sensor 118. Thus, the image sensor 118 captures the lightfiltered through the color filters on a grid of photosensitive pixelsand conveys red, green, or blue image information for each pixel toimage processing device 120.

Image processing device 120 is a computer processing system, such as adesktop computer, laptop computer, tablet computer, mobile telephone, orpurpose build computing device, which includes a processor, memory,communication interfaces, and other components typically found incomputer processing systems. One embodiment of a computing processingsystem is discussed in greater detail below in FIG. 5. In oneembodiment, image processing device 120 receives image data (e.g., red,green, or blue data for each pixel of image sensor 118) captured bycamera 114 of endoscope 110, which can be processed and displayed ondisplay 130 and/or stored in an imaging data store 140. For example,display 130 may render captured image data to a doctor, nurse, or othermedical professional that is analyzing image data captured by endoscope110. Furthermore, imaging data store 140 may store captured image datafor later analysis by a health care professional, such as storing imagedata captured during a medical procedure for further training a machinelearning model used by endoscope controller 125, storing image data to apatient's electronic medical records, etc.

In one embodiment, endoscope 110 utilizes programmable light source(s)112 to illuminate a scene during a medical procedure. As illustrated inFIG. 1, different types of tissue 180 (e.g., tissue A and tissue B) areimaged during a medical procedure using endoscope 110. In oneembodiment, the programmable light source(s) 112 include one or morediscrete lasers that are configurable by endoscope controller 125 atselected wavelengths and power levels when illuminating the sceneincluding the different tissues 180. Furthermore, the power of eachlaser can be independently modulated as determined by endoscopecontroller 125 thereby creating the independently programmable lightsource(s).

In one embodiment, camera 114 captures the light generated by theprogrammable light source(s) 112 and reflected from the differenttissues 180 in the scene using image sensor 118. In one embodiment,image sensor 118 includes a Bayer filter having a color filter arrayarranged over a grid of photosensors. Thus, every pixel of the camera's114 sensor 118 captures a function of the red, green, or blue lightreflected from a scene, such as each photosensor recording a differentweighted integral of the incident spectrum (e.g., Red=∫(I(λ)R(λ)r(λ)d,Green=∫(I(λ)R(λ)g(λ)d, and Blue=∫(I(λ)R(λ)b(λ)d). That is, the recordedred, green, or blue values at each pixel are a function of the spectrumof the programmable light source(s) 710 (e.g. I(λ)), the reflectivity ofthe tissue being imaged at each pixel 720 (e.g., R(λ)), and the Bayerfilter color functions for the associated pixel filter color (e.g., oneof r(λ), g(λ), or b(λ)), as illustrated in FIG. 7. The recorded value isa color value at each pixel, which is a Bayer filter color function 730value of the wavelength of the reflected light. As will be discussed ingreater detail below, the powers applied to the discrete programmablelight source(s) 112 by endoscope controller 125 are optimized such thatthe data captured by color pixels for red, green, and blue pixels of thefilter 118 optimally capture differences in reflectivity of differenttissue types. Endoscope controller 125 utilizes a trained machinelearning model, such as a trained neural network, to analyze thecaptured red, green, or blue color values at each pixel and classifyeach pixel as belonging to a specific tissue type or an unknown tissuetype. For example, based on differences in reflectivities of tissue Aand tissue B 180, endoscope controller's 125 machine learning modelbased analysis of the captured color data enables endoscope controllerto classify each pixel as belonging to tissue of type A, type B, orneither, based on a probabilistic analysis performed by the trainedmachine learning model that a pixel belongs to one of the classes oftissue.

In one embodiment, endoscope controller 125 generates visualization 182of the tissue types in real time during a medical procedure based on thedetermined classification probabilities, which is rendered on display130. In one embodiment, the visualization 182 of each tissue type duringsurgery includes a color overlay selectively displayed over each pixelof captured image data on display 130. The color overlay image caninclude a different color for each type of tissue rendered in real timeover the captured image data for a surgeon or other medicalprofessional. That is, for each classifiable tissue type, a differentcolor is assigned, and a probability mapping of each tissue type topixels of a captured image can govern the opacity of the color over theimage to, in embodiments, distinguish between different tissue types andprovide an indication of the likely accuracy of the determinedclassification. Thus, different color overlays are created for eachtissue type at each pixel of captured image data to differentiateregions of tissue type, increase contrast between tissue types, warn amedical professional using endoscope of the location of specific typesof tissue, etc. These color overlays can be rendered to create a singlevisualization 182 where different types of tissue are conspicuouslydifferentiated based on the boundaries of the color overlays. However,each color overlay can be toggled through by a surgeon or medicalprofessional in real time as image data is captured by endoscope 110.

FIG. 2 is a block diagram of one embodiment 200 of an endoscope 210 andan image processing device 250. Endoscope 210 and an image processingdevice 250 provide additional details for the endoscope 110 and imageprocessing device 120 discussed above. Furthermore, although not shown,image processing device may be communicably coupled to a display and/oran imaging data store as discussed above.

In one embodiment, the image processing device 250 includes endoscopecontroller 252 having an endoscope interface 255, a light enginecontroller 260, a machine learning model analysis engine 265, a displayrendering engine 290, an optional machine learning model training engine270, and a store of machine learning models 280. As discussed above inFIG. 1, the endoscope 210 includes one or more programmable lightsource(s) 212 (e.g., individually controllable lasers) and a camera 214having a lens 216 and an image filter 218 (e.g., a Bayer color filterover an array of photosensors/pixels). In one embodiment, the endoscope210 communicates with image processing device 250 via endoscopeinterface 255 over a wired or wireless communications link, as discussedabove in FIG. 1.

The image processing device 250, in embodiments, can be implemented in acomputing device, such as a desktop computer, laptop computer, tabletcomputer, purpose built computing appliance, video game console, mobiletelephone, as well as other computing devices. Endoscope interface 255is responsible for communicably coupling the endoscope controller 252 ofimage processing device 250 with endoscope 210 to enable endoscopecontroller 260 the ability to control the programmable light source(s)212 of endoscope 210, as well as to receive color pixel data captured bycamera 214 of endoscope 210.

In one embodiment, light engine controller 260 is responsible forconfiguring the programmable light source(s) 212 of endoscope. In oneembodiment, programmable light source(s) 212 include individuallyconfigurable light sources that illuminate a scene (e.g., discretewavelength laser light sources with modulatable power levels). In oneembodiment, light engine controller 260 configures each of theprogrammable light source(s) 212 to illuminate the scene at acorresponding power level. In one embodiment, the power level for eachof the programmable light source(s) 212 is determined by light enginecontroller 260 from a trained machine learning model (MLM) used by MLManalysis engine 265 when performing multi-class tissue typeclassification. A machine learning model, which is trained according tothe discussion below, is accessed by endoscope controller 125 todetermine power weighting values corresponding to power weightingfunctions found by, for example neural network optimization, to programthe intensities/power levels applied to the one or more programmablelight source(s) 212 by light engine controller 260.

Camera 214 captures an image of the scene as illuminated by programmablelight source(s) 212. More specifically, the light reflected from one ormore types of tissue in the scene passes through lens 216 and ontosensor 218. Sensor 218, in one embodiment, is a Bayer color filter thathas a matrix of red, green, and blue color pixels. The color valuescaptured by each pixel are provided to endoscope controller 252 for MLManalysis engine 265 to analyze using a selected machine learning modelcorresponding to the machine learning model used to configure theprogrammable light source(s) 212.

Prior to use of a machine learning model by MLM analysis engine 265,endoscope controller may obtain one or more MLM models trained by aremote system using the techniques discussed herein, and store them inMLM data store 280. Additionally, or alternatively, MLM training engine270 may perform a machine learning process to locally train one or moremachine learning models for performing multi-class tissue typeclassification or further refine an already trained MLM. In embodiments,each machine learning model utilized by MLM analysis engine 265 enablesdifferentiation between a set of two or more types of tissue.

In one embodiment, where endoscope controller 252 locally performsmachine learning model training, MLM training engine 270 is communicablycoupled with a hyperspectral imager (not shown) via hyperspectral imager(HSI) interface 275. In this embodiment, the hyperspectral imager isresponsible for supply training data to the MLM training engine 270 inthe form of measured reflectance of one or more tissue samples inresponse to a plurality of different discrete illumination wavelengths.In one embodiment, MLM training engine 265 trains a machine learningmodel, such as the neural network illustrated in FIG. 6, fordifferentiating tissue types of interest. For example, the machinelearning model of FIG. 6 can differentiate, after training, between aplurality of different tissue types, such as tissue types A though M.

In one embodiment, to train the machine learning model of FIG. 6, tissuesamples of the tissue types of interest, such as tissue types A throughM, are imaged with a hyperspectral imager (not shown). Hyperspectralimaging yields reflectance for a plurality of discrete wavelengths forevery pixel of image data captured by the hyperspectral imager. Sincethe tissues being imaged by the hyperspectral image are of known type(e.g., one or more samples of each of tissue types A through M), thedeep neural network machine learning model 610 can be iterativelytrained by MLM analysis engine 265 using any of the standard machinelearning techniques from the spectral reflectivity inputs of each pixel(e.g., red, green, and blue pixels), and the desired output being (e.g.,a vector of probabilities having a one for the known tissue type and azero for all other tissue types). As illustrated in FIG. 6, oneembodiment of a deep neural network machine learning model 610 includesa plurality of layers of weighted activation functions, where eachweighted activation function receives an input from a previous layer andcomputes an output that is provided to activation function(s) of otherlayers. The interconnected nodes, and their weighted computations, caninclude linear activation functions, non-linear activation functions, ora combination of both based on the classification to be performed andthe results of the classification process (e.g., whether lineartechniques yield expected results, or whether non-linear or combinedtechniques yield expected results). In one embodiment, the deep neuralnetwork machine learning model 610 includes a light engine intensities612 layer of activation functions, an RGB color value 614 layer ofactivation functions and zero or more zero or more additional layers ofMLM activation functions. The reflection input 602, light engineintensities 212, and RGB color values 614 essentially model the physicssurrounding the illumination of the tissue sample and the resultingreflection, with the remaining layers of activation functions enablingstatistical prediction from the modeled physics of the input and firsttwo machine learning model layers. That is, the training input and firsttwo layers of the deep neural network machine learning model 610calculate the values of red, green, and blue from the spectralreflectivities 614, while the remaining layers 616 of the neural networktake the red, green, and blue intensity values, as well as the weight ofall nodes after the first three layers, and transform them into thetissue type probability vector 620. The programmable light intensitiesas well as the weightings of all nodes after the first two layers aretherefore iteratively optimized by the machine learning process in orderto minimize training error for the samples of known tissue type.

During training, responsive to the input sample reflection data 602 andbased on a machine learning process, the light engine intensities 612layer of activation functions iteratively refine weightings applied tothe wavelengths of the illumination source (e.g., programmable lightsource(s) 212), iteratively refine the RGB color value 614 activationfunctions (e.g., computed red, green, or blue color values as a functionof the light engine intensities 612 and the input sample reflection602), and iteratively refine the zero or more additional layers of MLMactivation functions to minimize training error of the expectedprobability vector associated with a tissue of known type. That is, themachine learning process used by MLM analysis engine 265 to train themachine learning model iteratively adjusts the activation functions andweightings of each layer of activation function at each node of theneural network machine learning model 610 with each training iterationto minimize the error when obtaining a desired output having aprobability of 1 for the actual tissue type of the sample being imaged(e.g., the known tissue type), and a probability of 0 for all the othertissue types (e.g., other classifications not being imaged).

In embodiments, machine learning models, such as neural network machinelearning model 610, are scalable to any number of categories/types oftissue. After a sufficient number of tissue samples have been imaged ata sufficient number of wavelengths by a hyperspectral imager, and MLMtraining engine 270 has performed a corresponding number of iterativerefinements of the deep neural network machine learning model 610, theresulting model can be stored in MLM data store for later use by MLManalysis engine 265.

In embodiments, however, the deep neural network machine learning model610 may alternatively be trained by a remote system (not shown) and notMLM training engine 270. In this embodiment, the remote system providesendoscope controller 252 with the trained machine learning model foruser during medical procedures where distinguishing between tissue typesin the model are desired. In embodiments, the remote system can providea set of procedure specific trained machine learning models, a set ofdifferent multi-class classifiers for distinguishing between differentgroupings of tissue type, etc. for storage in MLM data store 280. Asanother example, MLM analysis engine 265 can request and receive aspecific multi-class tissue type classification model on aprocedure-by-procedure basis.

In another embodiment, MLM training engine 270 can receive training datafrom endoscope 210. In embodiments, this training data can be receivedin response to the training of a new machine learning model or inresponse to a refinement of an existing machine learning model. In thisembodiment, MLM training engine 270 initially receives highlighted,annotated, or otherwise differentiated areas of image data captured byendoscope 210 (e.g., a surgeon's real time annotations of image dataduring a medical procedure). For example, a cursor or touch input devicecan provide input for a display system coupled with an image processingdevice, which is displaying real-time image data of a scene captured byendoscope 210. In embodiments, the differentiated areas are received inreal-time with annotations for each area and the corresponding believedtissue type. MLM training engine 270 instructs light engine controller260 to individually iterate through each of the discrete programmablelight source(s) 212, illuminating the scene with each light source oneat a time and storing the corresponding image in a memory accessible toMLM training engine 270. The set of images corresponding to thedifferent light sources yields partial spectral reflectivities for eachpixel of captured image data. In one embodiment, the partial spectralreflectivities are fed into the machine learning model for iterativelytraining a machine learning model, such as a deep neural network machinelearning model, as discussed above. In one embodiment, a new machinelearning model may be trained using the partial spectral reflectivities.In another embodiment, an existing trained machine learning model (e.g.a model trained using hyperspectral imaging), may be refined usingreal-time surgical annotations.

After training a machine learning model using one of the techniquesdiscussed above, or after acquiring a trained machine learning model,endoscope controller 252 utilizes the trained machine learning model forperforming real time multi-class tissue type classification duringmedical procedures, and generating appropriate visualizations. In oneembodiment, MLM analysis engine 265 performs a pixel wise machinelearning model analysis in order to classify each pixel as belonging toone of a plurality of different classes of tissue types. In embodiments,color information captured by each pixel is fed into an appropriateintermediate layer of the machine learning model being used by MLManalysis engine 265 (e.g., layer 614 of machine learning model 610). Theintermediate layer of the machine learning model takes as input thecolor value of a Bayer color filter pixel, performs a series of machinelearning model computations using the color value as an input, andoutputs a probability vector with probabilities that the pixel belongsto one of the various potential classes of tissue. That is, during amedical procedure, the recorded red, green, or blue color value for eachpixel of filter 218 is fed by MLM analysis engine 265 into theappropriate intermediate layer of a trained machine learning model(e.g., the RGB color value layer 614 of the trained deep neural network610 illustrated in FIG. 6). The machine learning model 610 then outputsa probability vector to MLM analysis engine 265 that provides aprobability that imaged tissue at the pixel is one of the differentclasses of tissue, for each pixel of the image. The probability vectorsfor the pixels of captured image data enable MLM analysis engine 265 toclassify each pixel of an image as belonging to one of the multipleclassifications (e.g., belonging to tissue of Type A or Type B in FIG.1, or as belonging to any of tissue types A through M in FIG. 6).Although full spectral information can be used to train the machinelearning model when using a hyperspectral imager, as discussed ingreater detail above, the full spectral information is not utilized byMLM analysis engine 265 when using the trained machine learning model toclassify different tissue types in image data. Rather, that informationis encoded into the one or more programmable light source(s) 112 usingthe machine learning model as trained. This physical preprocessingprovides a surgeon or other medical professional utilizing endoscope 210and image processing device 250 the benefit of hyperspectral information(e.g., used to train the machine learning model), without needing aslow, bulky, and costly hyperspectral imager in endoscope 210 (e.g.,applying the machine learning model using color information).

In embodiments, MLM analysis engine 265 performs machine learning modelbased analysis of color information captured for each pixel of imagedata in real time as image data is captured by camera 214, and providesthe analysis results to display rendering engine 290. Display renderingengine 290 receives the analysis results, including the probabilityvectors for each pixel of image data. In one embodiment, the probabilityvectors are utilized to generate one or more visualization(s) thatenhance captured image data rendered on a display device (not shown)coupled with image processing device. The visualization(s) can includeenhancing contrast between two regions having tissues of different types(e.g., enhancing contrast between nerve and cancer tissue), generatingcolor overlays for different classes of tissue types (e.g., coloroverlays over fat, muscle, and intestinal tissue), generating a noticefor unknown zones of tissue type (e.g., a zone for which a probabilityvector is not able to predict any tissue type of interest within athreshold degree of accuracy), generating warnings when a medical toolencroaches on a zone having a specific tissue type (e.g., when a medicaltool approaches a threshold distance to nerve tissue), etc. Thevisualizations are generated in real time or near real time as imagedata is captured by camera 214. Thus, the visualizations providevaluable real-time feedback and tissue type differentiation to medicalprofessionals during a medical procedure.

FIG. 3 illustrates one embodiment of a process 300 for using a machinelearning model to perform multiclass classification of images using aprogrammable light source. The process is performed by processing logicthat may comprise hardware (circuitry, dedicated logic, etc.), software(such as is run on a general purpose computer system or a dedicatedmachine), firmware, or a combination. In one embodiment, the process isperformed by an endoscope and an image processing device (e.g.,endoscope 110 or 210, and image processing device 120 or 250).

Referring to FIG. 3, processing logic begins by accessing a trainedmulti-class tissue type classification machine learning model(processing block 302). As discussed herein, one or more trained machinelearning models for differentiating between tissue types based on colorinformation may be obtained from a remote system and/or trained byprocessing logic. Each machine learning model may be a trained neuralnetwork that differentiates between different sets of tissue types, andmay be used for different or select procedures. For example, machinelearning models relevant to abdominal procedures, cardiac procedures, orother types of procedures can provide for tissue type classification forthe types of tissue likely to be encountered during the respectiveprocedures. Furthermore, processing logic can access one of the machinelearning models based on user selection prior to usage of an endoscopein the associated procedure.

Processing logic configures one or more programmable light source(s) ofan imaging system based on a weighting of light source(s) in the machinelearning model (processing block 304). In one embodiment, the weightingscorresponding with weighting extracted from the selected machinelearning model. For example, the weightings can correspond withweightings applied to light engine intensities as the result of thetraining of the selected machine learning model (e.g., weightingsapplied to activation functions 612). In embodiments, processing logicpowers each light programmable light source according to its respectiveweighting so that the combined weightings applied to the programmablelight sources match the weightings extracted from the machine learningmodel.

Processing logic illuminates a scene with the configured light source(s)(processing block 306) Processing logic then captures an image of thescene including one or more types of tissue depicted in the image data(processing block 308). As discussed herein, the light sourceilluminates the scene by shining one or more laser lights at discreteillumination wavelengths and configured power levels. An image sensor,such as Bayer color filter having an array of photosensors with a colorgrid overlay, then captures the light reflected from the tissue(s) inthe scene.

For each pixel of captured image data, the pixels corresponding topixels of an image filter of a camera, processing logic inputs colorinformation from the captured image data into an intermediate layer ofthe machine learning model to classify that pixel as belonging todifferent tissue types (processing block 310). In one embodiment, theintermediate layer is an intermediate layer of a trained neural networkmachine learning model that performs activation function calculations onpixel color value, which are in turn used by the remaining layers of thetrained neural network machine learning model to compute a probabilityvector associated with that pixel. As discussed herein, the probabilityvector includes probabilities that the pixel belongs to each potentialtype of tissue. When the probability exceeds a threshold value (e.g.,85%, 90%, 99%, etc.), processing logic concludes that the pixel of imagedata has captured image data of the corresponding tissue type. This canbe repeated for each pixel so that the entire image, or a subset of animage, can be classified according to tissue type captured therein.

Processing logic utilizes the classifications to render a visualizationof the captured image data including a visual differentiation betweentissues determined to belong to different types (processing block 312).In embodiments, the differentiation between tissue types in the renderedvisualization can include contrast enhancement between tissue types ofinterest. In another embodiment, color overlays may be generated foreach tissue type of interest, with opacity of each color overlayadjusted to account for the computed probability (e.g., the higher theprobability, the more opaque a region is rendered in the visualization).In embodiments, the color overlays and/or contrast enhancements can beselectively activated and/or deactivated by the medical professional(s)performing the procedure. Furthermore, different visualizations can bestepped through, such as by turning on/off individual color overlays,enhancing contrast of selected types of tissue, etc. Each of thevisualizations and the control of the visualization provide a medicalprofessional improved imaging of a scene, which is valuable since themedical professionals typically perform medical procedures withendoscopes without actually seeing the scene.

FIG. 4 is a flow chart illustrating one embodiment of a process 400 fortraining a machine learning model for multiclass classification ofimages using a programmable light source. The process is performed byprocessing logic that may comprise hardware (circuitry, dedicated logic,etc.), software (such as is run on a general purpose computer system ora dedicated machine), firmware, or a combination. In one embodiment, theprocess is performed by an endoscope and an image processing device(e.g., endoscope 110 or 210, and image processing device 120 or 250), aremote processing system, or a combination of systems, as discussedherein.

Referring to FIG. 4, processing logic begins by initiating training of amachine learning model (MLM) for performing multi-class tissue typeclassification (processing block 402). In embodiments, the training isinitiated before a medical procedure is performed. The training caninclude training a new model using hyperspectral imaging to image aplurality of tissue samples at a plurality of illumination wavelengths.The training can also include training a new model using theillumination sources of an endoscope, and tissue type selects made by amedical professional. Additionally, the training can include acombination of both training techniques where an existinghyperspectrally trained machine learning model is further refined withreal-time tissue type selection.

Processing logic measures a spectral reflectance of a tissue sample of atissue type of interest at one of a plurality of discrete illuminationwavelengths to determine reflectance of the tissue sample as a functionof that illumination wavelength (processing block 404). In oneembodiment, the spectral reflectance corresponds with measurements takenfor a plurality of discrete illumination wavelengths generated by ahyperspectral imager. In another embodiment, the spectral reflectancecorresponds with partial spectral reflectivity measurements taken foreach programmable light source of an endoscope.

Processing logic inputs the measured spectral reflectance for the tissuetype into a MLM training process for the machine learning model(processing block 406). As discussed herein, the input is used toiteratively train the machine learning model by inputting the measuredspectral reflectance values into the machine learning model for a knowntissue type. The training methods are discussed in greater detail above.

Processing logic then determines if there are more wavelengths for whichspectral reflectance data is desired (processing block 408) and if thereare more tissue type samples of the same and/or different tissue type(processing block 410). If either of these processing blocks is true,the process returns to processing block 404 to measure spectralreflectance and further train the machine learning model. Inembodiments, the machine learning model is iteratively trained using aplurality of different wavelengths of illumination of each tissue sampleand a plurality of different samples of the same and different type.However, when there is no more training data to be obtained and inputinto the machine learning model training process, the process ends.

FIG. 5 is one embodiment of a computer system that may be used with thepresent invention. The computer system may provide the functionality ofthe image processing systems discussed above. Furthermore, it will beapparent to those of ordinary skill in the art, however, that otheralternative systems of various system architectures may also be used.

The computer system illustrated in FIG. 5 includes a bus or otherinternal communication means 515 for communicating information, and aone or more processor(s) 510 coupled to the bus 515 for processinginformation. The system further comprises a random access memory (RAM)or other volatile storage device 550 (referred to as memory), coupled tobus 515 for storing information and instructions to be executed byprocessor(s) 510. Memory 550 also may be used for storing temporaryvariables or other intermediate information during execution ofinstructions by processor 510. The system also comprises a read onlymemory (ROM) and/or static storage device 520 coupled to bus 515 forstoring static information and instructions for processor 510, and adata storage device 525 such as a magnetic disk or optical disk and itscorresponding disk drive. Data storage device 525 is coupled to bus 515for storing information and instructions.

The system may further be coupled to a display device 570, such as alight emitting diode (LED) display, a liquid crystal display (LCD),etc., coupled to bus 515 through bus 565 for displaying information to acomputer user, such as a medical professional utilizing the imageprocessing system during a medical procedure. An alphanumeric inputdevice 575, including alphanumeric and other keys, may also be coupledto bus 515 through bus 565 for communicating information and commandselections to processor 510. An additional user input device is cursorcontrol device 580, such as a mouse, a trackball, stylus, or cursordirection keys coupled to bus 515 through bus 565 for communicatingdirection information and command selections to processor 510, and forcontrolling cursor movement on display device 570.

Another device, which may optionally be coupled to computer system 500,is a communication device 590 for accessing other nodes of a distributedsystem via a network. The communication device 590 may include any of anumber of commercially available networking peripheral devices such asthose used for coupling to an Ethernet, token ring, Internet, or widearea network. The communication device 590 may further be a null-modemconnection, or any other mechanism that provides connectivity betweenthe computer system 500 and the outside world. Note that any or all ofthe components of this system illustrated in FIG. 5 and associatedhardware may be used in various embodiments of the present invention.

It will be appreciated by those of ordinary skill in the art that anyconfiguration of the system may be used for various purposes accordingto the particular implementation. The control logic or softwareimplementing the present invention can be stored in memory 550, datastorage device 525, or other storage medium locally or remotelyaccessible to processor(s) 510.

It will be apparent to those of ordinary skill in the art that thesystems, methods, and processes described herein can be implemented assoftware stored in memory 550 or read only memory 520 and executed byprocessor(s) 510. This control logic or software may also be resident onan article of manufacture comprising a computer readable medium havingcomputer readable program code embodied therein and being readable bythe data storage device 525 and for causing the processor(s) 510 tooperate in accordance with the methods and teachings herein.

The present invention may also be embodied in a handheld or portabledevice, such as a tablet computer system, laptop computer system,smartphone, smart glasses, etc., containing a subset of the computerhardware components described above. For example, the handheld devicemay be configured to contain only the bus 515, the processor(s) 510, andmemory 550 and/or 525. The handheld or portable device may also beconfigured to include a set of buttons or input signaling componentswith which a user may select from a set of available options. Thehandheld or portable device may also be configured to include an outputapparatus such as a liquid crystal display (LCD) for displayinginformation to a user of the handheld or portable device. Conventionalmethods may be used to implement such a handheld or portable device. Theimplementation of the present invention for such a device would beapparent to one of ordinary skill in the art given the disclosure of thepresent invention as provided herein.

The present invention may also be embodied in a special purposeappliance including a subset of the computer hardware componentsdescribed above. For example, the appliance may include a processor(s)510, a data storage device 525, a bus 515, and memory 550, and onlyrudimentary communications mechanisms, such as a small touch-screen thatpermits the user to communicate in a basic manner with the device. Ingeneral, the more special-purpose the device is, the fewer of theelements need be present for the device to function.

The processes explained above are described in terms of computersoftware and hardware. The techniques described may constitutemachine-executable instructions embodied within a tangible ornon-transitory machine (e.g., computer) readable storage medium, thatwhen executed by a machine will cause the machine to perform theoperations described. Additionally, the processes may be embodied withinhardware, such as an application specific integrated circuit (“ASIC”) orotherwise.

A tangible machine-readable storage medium includes any mechanism thatprovides (i.e., stores) information in a non-transitory form accessibleby a machine (e.g., a computer, network device, personal digitalassistant, any device with a set of one or more processors, etc.). Forexample, a machine-readable storage medium includesrecordable/non-recordable media (e.g., read only memory (ROM), randomaccess memory (RAM), magnetic disk storage media, optical storage media,flash memory devices, etc.).

The above description of illustrated embodiments of the invention,including what is described in the Abstract, is not intended to beexhaustive or to limit the invention to the precise forms disclosed.While specific embodiments of, and examples for, the invention aredescribed herein for illustrative purposes, various modifications arepossible within the scope of the invention, as those skilled in therelevant art will recognize.

These modifications can be made to the invention in light of the abovedetailed description. The terms used in the following claims should notbe construed to limit the invention to the specific embodimentsdisclosed in the specification. Rather, the scope of the invention is tobe determined entirely by the following claims, which are to beconstrued in accordance with established doctrines of claiminterpretation.

What is claimed is:
 1. A method of training a machine learning model toidentify one or more types of tissue included in different tissue types,the method comprising: illuminating one or more tissue samples for eachof the one or more types of tissue; measuring spectral reflectivities ofthe one or more tissue samples at a plurality of discrete illuminationwavelengths to determine a reflectance of the one or more tissue samplesas a function of illumination wavelength; iteratively training themachine learning model, using the spectral reflectivities as inputs tothe machine learning model, for probabilistic identification of a tissuetype based on color image data from a camera, wherein the machinelearning model includes an artificial neural network having a pluralityof layers of weighted transfer functions, including: a first layer thatmultiples the inputted spectral reflectivities by a spectrum of anillumination source; a second layer that multiplies values computed inthe first layer one or more color functions of the camera; and at leastone additional layer that receives the multiplied values computed by thesecond layer, and wherein an output of the machine learning model is avector of probabilities corresponding to the probabilisticidentification of the tissue type.
 2. The method of claim 1, wherein thespectral reflectivities are measured with a hyperspectral imager.
 3. Themethod of claim 1, wherein the one or more color functions of the camerainclude a red color function, a blue color function, and a green colorfunction that each correspond to a respective filter of the camera,wherein the camera is not a hyperspectral imager, and wherein thetraining of the machine learning model with the spectral reflectivitiesconfigures weightings of at least the first layer to correspond to aconfiguration of one or more programmable light sources for theprobabilistic identification of the tissue type from the color imagedata provided by the camera.
 4. The method of claim 3, wherein theconfiguration corresponds to intensity or power of the one or moreprogrammable light sources.
 5. The method of claim 3, wherein theconfiguration provided by the machine learning model capturesdifferences in spectral reflectivity between the different tissues typesthat are within at least one of the red color function, the blue colorfunction, or the green color function of the respective filter of thecamera.
 6. The method of claim 3, further comprising refining themachine learning model with additional training, including: applying theconfiguration to the one or more programmable light sources based on atleast the weightings of the first layer of the machine learning model toilluminate a scene; receiving real-time user annotations of the imagedata of the scene that classify portions of the image data depicting theone or more types of tissue as belonging to one of the different tissuetypes; iteratively illuminate the scene with each of the one or moreprogrammable light sources and capturing images associated with eachilluminated version of the scene with the camera, wherein the capturedimages provide partial spectral reflectivities of the classifiedportions; and providing the partial spectral reflectivities as inputscorresponding to the input spectral reflectivities to iteratively trainthe machine learning model to provide the probabilistic identificationof the tissue type from the color image data.
 7. The method of claim 1,wherein the training of the machine learning model is specific to a typeof procedure.
 8. The method of claim 1, wherein the inputted spectralreflectivities, the first layer, and the second layer collectivelycalculate values of red, green, and blue intensity values associatedwith the one or more color functions, and wherein subsequent layers ofthe machine learning model, including the at least one additional layer,take the red, green, and blue intensity values and transform them intothe vector of probabilities based, at least in part, on weights of allnodes included in the subsequent layers.
 9. The method of claim 1,wherein the second layer is disposed between the first layer and the atleast one additional layer of the machine learning model.
 10. The methodof claim 1, wherein the training of the machine learning modeliteratively adjusts weighting for nodes included in each of theplurality of layers of the weighted transfer function for each trainingiteration.
 11. A non-transitory machine readable storage medium havinginstructions stored thereon, which when executed by a processing system,cause the processing system to perform a method of training a machinelearning model to identify one or more types of tissue included indifferent tissue types, comprising: illuminating one or more tissuesamples for each of the one or more types of tissue; measuring spectralreflectivities of the one or more tissue samples at a plurality ofdiscrete illumination wavelengths to determine a reflectance of the oneor more tissue samples as a function of illumination wavelength;iteratively training the machine learning model, using the spectralreflectivities as inputs to the machine learning model, forprobabilistic identification of a tissue type based on color image datafrom a camera, wherein the machine learning model includes an artificialneural network having a plurality of layers of weighted transferfunctions, including: a first layer that multiples the inputted spectralreflectivities by a spectrum of an illumination source; a second layerthat multiplies values computed in the first layer one or more colorfunctions of the camera; and at least one additional layer that receivesthe multiplied values computed by the second layer, and wherein anoutput of the machine learning model is a vector of probabilitiescorresponding to the probabilistic identification of the tissue type.12. The non-transitory machine readable storage medium of claim 11,wherein measuring the spectral reflectivities is performed with ahyperspectral imager.
 13. The non-transitory machine readable storagemedium of claim 11, wherein the one or more color functions of thecamera include a red color function, a blue color function, and a greencolor function that each correspond to a respective filter of the camerathat is not a hyperspectral imager, and wherein the training of themachine learning model with the spectral reflectivities configuresweightings of at least the first layer to correspond to a configurationof one or more programmable light sources for the probabilisticidentification of the tissue type from the color image data provided bythe camera.
 14. The non-transitory machine readable storage medium ofclaim 13, wherein the configuration corresponds to intensity or power ofthe one or more programmable light sources.
 15. The non-transitorymachine readable storage medium of claim 13, wherein the configurationprovided by the machine learning model captures differences in spectralreflectivity between the different tissues types that are within atleast one of the red color function, the blue color function, or thegreen color function of the respective filter of the camera.
 16. Thenon-transitory machine readable storage medium of claim 13, furthercomprising refining the machine learning model with additional training,including: applying the configuration to the one or more programmablelight sources based on at least the weightings of the first layer of themachine learning model to illuminate a scene; receiving real-time userannotations of the image data of the scene that classify portions of theimage data depicting the one or more types of tissue as belonging to oneof the different tissue types; iteratively illuminate the scene witheach of the one or more programmable light sources and capturing imagesassociated with each illuminated version of the scene with the camera,wherein the captured images provide partial spectral reflectivities ofthe classified portions; and providing the partial spectralreflectivities as inputs corresponding to the input spectralreflectivities to iteratively train the machine learning model toprovide the probabilistic identification of the tissue type from thecolor image data.
 17. The non-transitory machine readable storage mediumof claim 11, wherein the training of the machine learning model isspecific to a type of procedure.
 18. The non-transitory machine readablestorage medium of claim 11, wherein the inputted spectralreflectivities, the first layer, and the second layer collectivelycalculate values of red, green, and blue intensity values associatedwith the one or more color functions, and wherein subsequent layers ofthe machine learning model, including the at least one additional layer,take the red, green, and blue intensity values and transform them intothe vector of probabilities based, at least in part, on weights of allnodes included in the subsequent layers.
 19. The non-transitory machinereadable storage medium of claim 11, wherein the second layer isdisposed between the first layer and the at least one additional layerof the machine learning model.
 20. The non-transitory machine readablestorage medium of claim 11, wherein the training of the machine learningmodel iteratively adjusts weighting for nodes included in each of theplurality of layers of the weighted transfer function for each trainingiteration.